FCoE VN—port to FC N—port operations in an ethernet fabric

ABSTRACT

Ethernet Fabric devices all act as FCoE to FC gateways, so that the Ethernet Fabric itself is considered a virtual FCoE to FC gateway. Further, the Ethernet Fabric allows direct routing of FCoE packets from one FCoE device coupled to the Ethernet Fabric to another FCoE device coupled to the Ethernet Fabric.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. §119(e) of U.S.Provisional Patent Application Ser. No. 61/780,700 entitled “FCoEVN_port to FC N_port Operations in an Ethernet Fabric,” filed Mar. 13,2013, which is hereby incorporated by reference.

This application is related to patent application Ser. No. 14/206,946,filed concurrently herewith, entitled “FCoE VN_Port Virtualizer,” whichis hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to network devices.

2. Description of the Related Art

One limitation of Fibre Channel (FC) is the practical number of domainsor switches that can be present in a fabric. Another concern with FC isthe amount of time required to reconfigure the fabric on removal oraddition of a new switch. To address these points, a system and methodas described in U.S. Pat. No. 7,707,309, entitled “Isolation Switch forFibre Channel Fabrics in Storage Area Networks” was developed. In thatdesign a switch is connected to hosts in the usual manner but isconnected to an FC fabric using N_Port ID Virtualization (NPIV) ports,with one NPIV address for each connected host. The switch translatedaddresses between those provided to the host and those assigned usingthe NPIV process. U.S. Pat. No. 7,577,134, entitled “Port Expander forFibre Channel Fabrics in Storage Area Networks” improved on the designof the '309 patent by passing the NPIV addresses directly to theconnected hosts, removing the translation step.

Another concern in FC has been the relatively high cost of the host busadapters (HBAs) and the switches, particularly as compared to thecorresponding Ethernet network interface cards (NICs) and switches androuters. To that end a technology called Fibre Channel over Ethernet(FCoE) has been developed. Basically an FC packet is encapsulated in anEthernet packet and transferred through a data center Ethernet network.In an FCoE environment, a host or Enode uses either a converged networkadapter (CNA), which receives FC packets and encapsulates them itself,or an FCoE driver, which receives the FC packets, converts them to FCoEpackets and then provides those to a normal NIC. The FCoE packets areprovided to a Fibre Channel Forwarder (FCF), which is a device that isconnected to both the Ethernet network and an FC fabric. The FCFreceives FCoE packets, converts them to FC packets and then providesthem to the FC fabric using an E_Port or fabric port. The FCF acts as anormal FC switch, performing fabric login operations and the like andconsumes a domain. In the opposite direction, from storage device tohost, the FCF receives the FC packets from the storage unit, convertsthem to FCoE packets and provides them to the Ethernet network, wherethey are delivered to the host. The FCoE subsystem of the host, either aCNA or a driver and NIC, convert the FCoE packets to FC packet data,which the host then processes. While this approach addresses a costissue, it does nothing to address the number of switches problem of FCas each FCF consumes another domain.

One approach to address that concern has been the development of theFCoE to FC gateway, essentially the combination of an FCF and the abovementioned Port Expander. The FCoE to FC gateway connects to the FCfabric using NPIV ports, as discussed above. The FCF portion of thegateway terminates the FCoE network by providing a VF_Port. Internallythe FC packets are obtained but then they are provided to the NPIV portfor transmission through the FC fabric. The FC portion of the gatewaybehaves just as the NPIV port of the port expander, except that it isconnected internally to an FCoE port rather than an F_Port. Thisaddresses the domain count concern but still requires each FCoE packetto be routed through the Ethernet network to the specific FCoE to FCgateway, as only that one unit has the needed information to perform thenecessary packet operations.

In a different area, the number of hosts used in a data center has beenincreasing dramatically, particularly with the advent of virtual machine(VM) usage. As each VM consumes a MAC address and usually an IP address,transferring packets inside a data center has been problematic ascomplex router structures are needed to handle the very large number ofIP addresses, as only 256 IP addresses can be present on one subnet.These 256 IP addresses can easily be met by a single blade serverchassis. The very large number of addresses or connections, virtual orphysical, also creates fundamental routing problems as Ethernet uses aspanning tree protocol (STP) for routing to prevent loops. However,using STP results in many non-optimal routes and difficulty in providingload balancing and multipath benefits. To address these issues EthernetFabrics have been developed, primarily based on TRILL, to handle therouting at the layer 2 level and to allow load balancing, multipathingand optimal or shortest path routing. One such Ethernet Fabric is theVCS architecture provided by Brocade Communications Systems, Inc.

If an FCoE to FC gateway is used with an Ethernet Fabric, the singlepoint of connection, the FCoE to FC gateway, remains a problem and to anextent reduces the value of the basic improvements provided by theEthernet Fabric.

A problem that has developed in FCoE and has not been resolved despitemany attempts, is the handling of peer-to-peer FCoE communication. Forvarious reasons, in an FCoE environment that includes an FC fabric, allpackets must travel to the FC fabric. Thus, any FCoE packet intended foranother FCoE device on the same Ethernet network must still travel tothe FC fabric and then back. Much debate has occurred, but an acceptableproblem has not been determined.

SUMMARY OF THE INVENTION

Embodiments according to the present invention use an Ethernet Fabricfor two additional purposes. First, the Ethernet Fabric devices all actas FCoE to FC gateways, so that the Ethernet Fabric itself is considereda virtual FCoE to FC gateway. Second, the Ethernet Fabric allows directrouting of FCoE packets from one FCoE device coupled to the EthernetFabric to another FCoE device coupled to the Ethernet Fabric withouthaving to go to the FC fabric and back.

BRIEF DESCRIPTION OF THE FIGURES

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate an implementation of apparatusand methods consistent with the present invention and, together with thedetailed description, serve to explain advantages and principlesconsistent with the invention.

FIG. 1 is a block diagram of an FCoE to FC gateway.

FIG. 2 is a block diagram of the information flow in the FCoE to FCgateway of FIG. 1.

FIG. 3 is a block diagram of a network including an Ethernet Fabricaccording to the present invention.

FIG. 4 is a block diagram illustrating FCoE VN_Port to FC N_Portoperations according to the present invention.

FIG. 5 is a block diagram illustrating FC N_Port to FCoE VN_Portoperations according to the present invention.

FIG. 6 is a block diagram illustrating FCoE VN_Port to FCoE VN_Port ondifferent VCS RB operations according to the present invention.

FIG. 7 is a block diagram illustrating FCoE VN_Port to FCoE VN_Port onsame VCS RB operations according to the present invention.

FIG. 8 is a block diagram illustrating FIP Discovery operationsaccording to the present invention.

FIG. 9 is a block diagram illustrating FIP VLI operations according tothe present invention.

FIG. 10 is a block diagram of a first alternate network including anEthernet Fabric according to the present invention.

FIG. 11 is a block diagram of a second alternate network including anEthernet Fabric according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

As shown in FIG. 1, an FCoE Switch 102 in FC Gateway mode (FFG: FCoE FCGateway) connects to an FC fabric or storage area network (SAN) 104 andruns a limited lightweight stack of FC services without using additionaldomains, dedicated or shared, from the SAN 104. An FFG 102 terminatesFCoE virtual links and extends fabric services provided by the FC SAN104 to attached devices, all without requiring the use of new domains.

Besides domain preservation, an FFG 102 provides topology flexibility.FCoE ENode devices 106 can be moved between FC SAN switches 108 withoutre-cabling the device connections. Since one FFG 102 can connect tomultiple FC SAN switches 108, the devices connected to the FFG 102 canbe moved across these switches just by changing software userconfiguration.

An FFG 102 works as a pass-through-module between ENode devices 106attached to it and the FC SAN switch 108 to which it connects. The ENodedevices 106 do not directly log into the FFG 102, rather the FFG 102converts the FCoE Initialization Protocol YIN logins to FC logins andforwards the FC logins to the attached FC switch 108. The port typessupported on the FFG 102 are VF_Ports no and N_Ports 112. VF_Ports noare created logically on top of physical CEE interfaces and providelogin services to the attached devices and the FFG 102 forwards theselogins over the FC Links 114 to the FC switch 108. These FC links 114consist of N_Ports 112 on the FFG 102 connecting to F_Ports 116 on theFC fabric switch 108. There are no E_Ports on the FFG 102 in selectedembodiments. VF_Ports no on the FFG 102 support NPIV capability in theirown right, which provides potential for two levels of NPIV, one to theVF_Port no and one to the F_Port 116.

User defined mappings associate VF_Ports 110 with N_Ports 112 on the FFG102. Since N_Ports 112 have a 1:1 physical connection to F_Ports 116 onthe FC Fabric switch 108, this VF—N association on the FFG 102determines which fabric switch F_Port 116 an ENode device 106 logs into.An N_Port 112 on the FFG 102 logs into the FC fabric switch 108 F_Port116 as a base NPIV device and gets assigned an FC_ID. The ENode devices106 on the FFG 102 which login through this N_Port 112 are treated assubsequent NPIV logins on the FC switch 108 F_Port 116. Thus, the ENodedevices 106 obtain their FC_IDs from the FC switch 108 F_Port 116.

Device FIP F Port Logins (FLOGIs) received on VF_Ports no are trappedand sent to an FC fabric switch 108 as FDISCs (Discover F PortParameters). Thus, each fabric F_Port 116 receives an FLOGI sent by theFFG 102 N_Port 112 and then subsequent FDISCs sent on behalf of theENode devices 106. The FIP and FLOGI exchange over an FFG 102 isoutlined in FIG. 2.

FIG. 2 shows an FFG 102 with a Converged Enhanced Ethernet (CEE) chip118 with 24 CEE interfaces and an FC chip 120 which has 8 FC portsfunctioning as N_Ports 112. VF_Ports 110 are created on top of CEEinterfaces 122. Thus there are 24 VF_Ports 110 on the FFG 102 shown inFIG. 2. Interconnect 136 is a switching fabric that allows packettransfer between the various FC ports and CEE interfaces 122 asappropriate. In the preferred embodiment packets may transfer between anFC port and a CEE interface or between two CEE interfaces. CP 124 is thecore processor on the FFG 102. Packets may be forwarded to the CP 124from either the FC ports or the CEE interfaces and the CP 124 canprovide packets to the FC ports or the CEE interfaces using either theinterconnect 136 or through sideband connections, neither of which areshown for simplicity. A Gateway Daemon (GD) 126 is a user space orsimilar software module which implements and controls the FFG mode andruns on the CP 124. Routing table storage 138 is provided for use by theFC chip 120, the FCoE/CEE chip 118 and the CP 124. The tables in therouting table storage 138 are discussed in more detail below.

As shown in FIG. 2, FIP VLAN and FCF discovery and responses are trappedby the VF_Ports 110 and forwarded for handling by the GD 126 simulatingan FCF as indicated by dashed lines 128. In the next phase, as indicatedby the dashed lines 130, the device FIP FLOGIs/FDISCs are trapped on theVF_Ports and sent to the GD 126. The GD 126 converts FIP FLOGIs to FCFDISCs, as an initial FC FLOGI will have occurred on FG 102 startup;converts any FIP FDISCs to FC FDISCs; and sends the FC FDISCs to the FCfabric switch 108 over the appropriate mapped N_Ports 112 as indicatedby the dashed lines 132. This FIP FLOGI conversion involves creating anew FC packet for the FDISC and copying over the device FLOGI into it.Thus, the GD 126 on the FFG 102 maintains a device login database andperforms login translation. The FDISC responses from the FC fabricswitch 108 are trapped on the ingress N_Ports 112 by the GD 126. Thesemay need to be converted to FIP FLOGI or FIP FDISC responses, asnecessary. These are copied into new FC packets and sent to the ENodedevices 106 after conversion to a FIP FLOGI or FDISC response. An FCFDISC response from the FC switch 108 contains the FC_ID for the device,so the ENode device 106 now has a PID (Physical ID) assigned. Forexample, an FFG 102 N_Port 112 performs FLOGI to the FC fabric switch108 and receives a base FC_ID of 0x010100. All subsequent device loginsthrough this N_Port 112 receive FC-IDs as 0x010101, 0x010102 etc.

This FC FDISC response from the FC switch 108 is the trigger for settingup routes for the data path on the FFG 102. PLOGI and subsequent dataframes from the ENode device 106 use the data path (shown via dash anddot lines 134 in FIG. 2). Once an ENode device 106 has logged into theFC SAN 104, the FFG 102 simply works as a pass-through module for dataframes.

The data path utilizes Virtual Fabric Routing (VFR) within the switch.Data frames are moved between the CEE interfaces 122 and FC ports 120with FCoE encapsulation and de-capsulation performed in the process.

The FFG 102 runs a significantly reduced set of FC Fabric services andthese are managed by the GD 126. For example, an FC Name Server and anFC Management Server do not run on FFG 102. The GD 126 on the FFG 102communicates with Fabric Services on an FC Switch 108 using FC-GSexchanges such as GMAL, GFN, RPAB, etc.

The FFG 102 provides graceful Failover/Failback for devices if the FClinks 114 go down. The FFG 102 provides several value-added services ifthe peer FC switch 108 supports these, such as trunking the N_Ports, QoSsupport and WWN-based login balancing.

This is a general overview of the basic operation and functioning of anFFG that presents VF_Ports to the ENode devices and N_Ports to the FCfabric.

FIGS. 3-9 illustrate Ethernet Fabric operations according to the presentinvention. The preferred embodiment uses the VCS architecture fromBrocade Communications Systems, Inc. More details on the VCSarchitecture are provided in U.S. Patent Application Publication Nos.2011/0299391, 2011/0286357, 2011/0268125, 2011/0299535, 2011/0268120,and 2011/0292947, which are hereby incorporated by reference.

In the preferred embodiment the Ethernet Fabric 302 provides:

Support for NPIV-based N_Port to F_Port bridging between FCoE devicesattached to the Ethernet Fabric and FC SAN attached devices,

Access to FCoE and FC targets from a single ENode interface,

Direct routing of FCoE to FCoE traffic without requiring a “trombone”from the Ethernet Fabric to the FC fabric and back to the EthernetFabric again, and

The preferred embodiment permits the Ethernet Fabric 302 to be viewed asa single NPIV FCoE to FC Gateway (FFG). In addition, from the FCoEdevice point of view, the Ethernet Fabric 302 appears as a singlelogical FCF. All Ethernet Fabric switches, also known as RBridges orRBs, 304 use the same external FCF MAC address.

All FCoE devices 306 log into, and are members of, the FC SAN fabric308. There is no separate, distinct FCoE SAN fabric. The Ethernet Fabric302 handles the FIP protocol processing and the FCoE encapsulationbetween the Ethernet Fabric 302 and the FC fabric 308, but all SANfabric services are provided by the FC SAN fabric 308 in the preferredembodiment.

Traffic between FCoE VN_Ports and FC N_Ports as shown by path 310 isbridged by an FFG switch functionality 314 embedded in an RBridge orEthernet fabric switch connected directly to the FC SAN 308 connected tothe FC fabric 308 via an FC N_Port to FC F_Port link. However, trafficbetween FCoE VN_Ports and FCoE VN_Ports is not forwarded to the FCFabric for routing; it is TRILL forwarded directly from the ingressEthernet Fabric switch 304 to the egress Ethernet Fabric switch 304 asshown by path 312 in FIG. 3. The RBs or Ethernet Fabric switches areinterconnected using Ethernet Fabric ports.

For the sake of clarity, we need to distinguish between the two types ofEthernet Fabric switches that may exist in this model. An EthernetFabric switch (RB for RBridge) that includes an Ethernet Fabric FCN_Port to FC F_Port (Ethernet Fabric-FC link) connection is called aG-RB switch, for Gateway-RBridge, in this document (G-RB1 304 in FIG. 3)as they contain embedded FFG switch functionality. These G-RB switchesmay be at the access edge of the Ethernet Fabric (Top-of-rack—TOR), atan aggregation layer (middle-of-rack or end-of-row—MOR, EOR), orattached to existing edge and/or aggregation level RBs as required.These G-RB switches support the following standard FCoE-FC gatewayservices including:

FCoE device login service;

Login of G-RB N_Port to FC fabric;

Conversion of ENode FIP FLOGI and FDISC to FC FDISC;

Mapping of incoming FCoE frames to N_Port uplinks based on FC frame'sSID Domain, Area;

FCoE encapsulation service, which is conversion of FCoE packets to FCframes and visa-versa; and

FIP discovery and VLAN services for FCoE devices attached to thisswitch.

In addition, G-RB switches provide the following additional services:

Ethernet Fabric-FC link management;

Via Ethernet Name Server (eNS) distribution services, which is used tomaintain coherency of information among the various RBs as discussed inPublication No. 2011/0299535 incorporated above, notify all other RBs oflink establishment, status, etc. (Ethernet Fabric-FC link is identifiedby FC SAN Domain, Area); and

Notify all other RBs of Ethernet Fabric-FC login count.

For deployments where the G-RBs are not deployed at the first EthernetFabric hop (TOR), RBs in the Ethernet Fabric that do not support aEthernet Fabric-FC link (RB2 304 and RB3 304 in FIG. 3) provide thefollowing services:

FIP discovery and VLAN services for FCoE devices attached to thisswitch;

FIP login redirection to the appropriate G-RB; and

Via the eNS distribution service, notify all other RBs of FCoE VN_Portadditions and deletions.

At a high level, the forwarding logic for a received FCoE frame is:

Promote to L3;

Perform 24 bit lookup on FC DID;

Perform 16 bit (Domain, Area) lookup on FC SID;

If (24 bit lookup=hit):

-   -   Result=Forward to egress RB (or egress interface if egress        RB=self);

Else (if 16 bit lookup=hit):

-   -   Result=Forward to egress G-RB (or egress interface if egress        G-RB=self);

Else:

-   -   Drop packet.

In order to facilitate the direct forwarding of FCoE to FCoE trafficfrom ingress RB to egress RB in the Ethernet Fabric, each RB in theEthernet Fabric has knowledge of every FCoE VN_Port attached to fabric,similar to the Ethernet Fabric model for L2 Ethernet MAC addresses.

When an FCoE Enode device successfully logs in to the FC SAN, the egressRB traps the FIP login response and updates all RBs of the new device'sFPMA MAC using the eNS distribution service. This eNS update includes aflag indicating that this is an FCoE VN_Port MAC, and rather thaninstalling an entry in the L2 tables, each RB installs a 24 bit FCoE DAentry using the low order three bytes of the VN_Port FPMA MAC address asthese correspond to the FC PID of the VN_Port. This allows any FCoEframe ingressing on any RB in the fabric, destined to the VN_Port, to bedirectly forwarded to the egress RB, the RB to which the VN_Port isattached.

Upon establishment of an N_Port to F_Port link from an RB in theEthernet Fabric to the FC SAN (an Ethernet Fabric-FC link) the RBnotifies all RBs in the Ethernet Fabric of the link establishment viathe eNS distribution service. The link is identified by the FC Domainand Area assigned by the FC fabric to the G-RB N_Port in response to theG-RB FLOGI. Each RB installs an FCoE ACL entry to facilitate a 16 bit FCSID Domain, Area based lookup with the result being forwarded to the RBowning the N_Port to F_Port link, or to the egress interface if this isthe owning RB. It is noted that the lookup to determine if the packet isdestined to a VN_Port or ENode connected to the Ethernet Fabric hasprecedence over these lookups which route the packet to the G-RB.

With these two sets of entries, lower 24 bits of FCoE DA for eachconnected ENode and 16 bit Domain, Area for each connected N_Port, addedto the normal Ethernet Fabric routing tables, in the Ethernet NameServer and other tables in the switch as described in Publication No.2011/0299535 incorporated above, all FCoE packets from ENodes or FCpackets from the FC fabric can be properly routed to the correct egressport of the Ethernet Fabric.

FIGS. 4 to 7 illustrate data flow for four cases, respectively FCoE hostto FC target, FC target to FCoE host, FCoE host to FCoE target on adifferent RB and FCoE host to FCoE target on the same RB.

Referring to FIG. 4, an exemplary network architecture is illustrated.An Ethernet Fabric 452 is connected to an FC SAN 454. The EthernetFabric 452 includes various interconnected switches. Switch G-RB1 456 isconnected to switch G-RB2 458. Switch G-RB1 456 and switch G-RB2 458include FFG functionality as indicated by the G-designation. SwitchG-RB1 456 and switch G-RB2 458 also act as RBridges in the EthernetFabric as indicated by the RB designation. Switches RB3 460, RB4 462 andRB5 464 are each connected to each of switches G-RB1 456 and G-RB2 458to form the Ethernet Fabric 452.

The FC SAN 454 is formed by FC switches 466, 468, 470 and 472, eachswitch including exemplary domain values. FC switches 466 and 468 areshown as each having two ports for connection to the Ethernet Fabric452, with port 1 being connected to switch G-RB1 456 and port 2 beingconnected to switch G-RB2 458. An FCoE host H1 474 is shown connected toswitch RB3 460, while FC target T1 476 is shown as connected to switch470, which has domain 12. This basic illustrated architecture is used invariations in FIGS. 4-9 to illustrate packet flow in various situations.

In FIG. 4 data flow from FCoE host H1 474 to FC target T1 476 isillustrated. Assume FCoE host H1 474 is logged into FC Domain 11 468,port (area) 2. Also assume there is a single external FCF MAC for theEthernet Fabric 452 which is programmed as a “Ext FCF MAC” on allRBridges in the Ethernet Fabric 452. Further assume that the EthernetFabric switches 456-464 each use an individual, internal FCF MAC valuefor ingress RBridge to egress RBridge/AG FCoE traffic. FCoE host H1 474provides an FCoE packet to switch RB3 460 as illustrated by the arrow A402. The FCoE packet has the following values: DA: Ext FCF MAC, SA: H1VN FPMA MAC, VID: FCoE VLAN, DID: T1 FCID and SID: H1 FCID.

Switch RB3 460 performs the following processing on the packet:

1. Promote to L3 (Because DA=Ext FCF MAC). The promotion to L3 resultsin using different fields for routing, in this case either the DID orSID fields in the embedded packet.

2. 24 bit DID lookup=miss (because the destination is not an FCoE deviceconnected to the Ethernet Fabric 452)

3. 16 bit SID lookup=hit (because it is a known FC ENode device),result=forward to switch G-RB2 458 (because of routing table entry)

4. TRILL forward to switch G-RB2 458 (normal Ethernet Fabric operation)

The Ethernet Fabric packet is forwarded from switch RB3 460 to switchG-RB2 458 as illustrated by arrow B 404. The packet has the followingvalues: TRILL DA=G-RB2, TRILL SA=RB3, Inner DA=G-RB2 int FCF MAC, InnerSA=RB3 int FCF MAC, VID=FCoE VLAN, DID=T1 FCID and SID=H1 FCID

Switch G-RB2 458 performs the following processing on the packet:

1. Terminate TRILL (because TRILL DA=G-RB2)

2. Promote to L3 (because DA=My Int FCF MAC)

3. 24 bit DID lookup=miss (because the destination is not an FCoE deviceconnected to the Ethernet Fabric 452)

4 16 bit SID lookup=hit (because it is a known FCoE ENode device),result=forward to FC (EGID) port (because of routing table entry).

The FFG functionality or entity embedded in switch G-RB2 458decapsulates the FCoE packet to produce the FC packet. The FC packet isprovided from switch G-RB2 458 to port 2 of switch 468 as illustrated byarrow C 406. The FC packet has the following values: DID=T1 FCID andSID=H1 FCID. Flow inside the FC SAN 454 is as normal and is notillustrated.

In FIG. 5 data flow from FC target T1 476 to FCoE host H1 474 isillustrated. Again assume FCoE host H1 474 is logged into FC domain 11,port (area) 2. Normal FC forwarding occurs on the FC packet from FCtarget T1 476 to FC switch 468, domain 11, port 2. Because FC switch 468assumes the FCoE host H1 is connected through port 2, the FC packetflows from FC switch 468 to switch G-RB2 458 as shown by arrow A 502.The FC packet has the following values: DID=H1 FCID and SID=T1 FCID. TheFFG functionality embedded in switch G-RB2 458 encapsulates the FCpacket to produce the FCoE packet with a DA of switch G-RB2 458 Int FCFMAC address and a null SA address as the RB function will supply aproper SA on transmission.

Switch G-RB2 458 performs the following processing on the packet:

1. Promote to L3 (because DA=My Int FCF MAC)

2. 24 bit DID lookup=hit (because the destination is an FCoE deviceconnected to the Ethernet Fabric 452), result=forward to RB3 460(because of the routing table entry)

The Ethernet Fabric packet is provided from switch G-RB2 458 to switchRB3 460 as illustrated by arrow B 504. The Ethernet Fabric packet hasthe following values: TRILL DA=RB3, TRILL SA=G-RB2, Inner DA=RB3 int FCFMAC, Inner SA=G-RB2 int FCF MAC, VID=FCoE VLAN, DID=H1 FCID and SID=T1FCID.

Switch RB3 460 performs the following processing:

1. Terminate Trill (because TRILL DA=RB3)

2. Promote to L3 (because DA=My Int FCF MAC)

3. 24 bit DID lookup=hit (because the destination is an FCoE deviceconnected to the Ethernet Fabric 452), result=forward to H1 (EGID) port(because of the routing table entry)

The FCoE packet is provided from switch RB3 460 to FCoE host H1 474illustrated by arrow C 506. The FCoE packet has the following values:DA: H1 VN FPMA MAC, SA: Ext FCF MAC, VID: FCoE VLAN, DID: H1 FCID andSID: T1 FCID.

In FIG. 6 data flow from FCoE host H1 474 to FCoE target T3 478, an FCoEtarget directly connected to the Ethernet Fabric 452. The FCoE host H1474 provides a packet to switch RB3 460 as illustrated by arrow A 602.The FcoE packet has the following values: DA: Ext FCF MAC, SA: H1 VNFPMA MAC, VID: FCoE VLAN, DID: T3 FCID and SID: H1 FCID.

Switch RB3 460 processes the FCoE packet as follows:

1. Promote to L3 (because DA=Ext FCF MAC)

2. 24 bit DID lookup=hit (because the destination is an FCoE deviceconnected to the Ethernet Fabric 452), result=forward to switch RB5 464(because of the routing table entry)

3. TRILL forward to RB5

The TRILL packet from switch RB3 460 flows to switch G-RB1 456 as shownby arrow B 604 and then to switch RB5 as shown by arrow C 606. This isnormal TRILL forwarding inside the Ethernet Fabric 452. The TRILL packethas the following values: TRILL DA=RB5, TRILL SA=RB3, Inner DA=RB5 intFCF MAC, Inner SA=RB3 int FCF MAC, VID=FCoE VLAN, DID=T3 FCID and SID=H1FCID.

Upon receipt of the TRILL packet switch G-RB1 performs normal TRILLforwarding. Upon receipt of the TRILL packet switch RB5 464 performs thefollowing processing:

1. Terminate TRILL (because TRILL DA=RB5)

2. Promote to L3 (because DA=RB5 int FCF MAC)

3. 24 bit SID lookup=hit (because the destination is an FCoE deviceconnected to the Ethernet Fabric 452), result=forward to T1 (EGID) port(because of the routing table entry)

The FCoE packet flows from switch RB5 464 to FCoE target T1 478 as shownby arrow D 608. The FCoE packet has the following values: DA: T3 VN FPMAMAC, SA: Ext FCF MAC, VID: FCoE VLAN, DID: T3 FCID and SID: H1 FCID.

In FIG. 7 data flow from FCoE host H1 474 to FCoE target T2 480connected to the same switch in the Ethernet Fabric 452 is illustrated.The FCoE packet flows from FCoE host H1 474 to switch RB3 460 as shownby arrow A 702. The FCoE packet has the following values: DA: Ext FCFMAC, SA: H1 VN FPMA MAC, VID: FCoE VLAN, DID: T2 FCID and SID: H1 FCID.

Switch RB3 460 processes the packet as follows:

1. Promote to L3 (because DA=Ext FCF MAC)

2. 24 bit SID lookup=hit (because the destination is an FCoE deviceconnected to the Ethernet Fabric 452), result=forward to T2 (EGID) port(because of the routing table entry)

The FCoE packet flows from switch RB3 460 to FCoE target T2 480 as shownby arrow B 704. The FCoE packet has the following values: DA: T2 VN FPMAMAC, SA: Ext FCF MAC, VID: FCoE VLAN, DID: T2 FCID and SID: H1 FCID.

FIG. 8 illustrates the FIP VLAN and FIP FCF discovery operations. ForFIP VLAN discovery, the FIP Request is provided from FCoE host H1 474 toswitch RB3 460 as shown by arrow A 802. The FIP Request packet has thefollowing values: DA: ALL_FCFs_MAC, SA: H1 ENode MAC and VID: Any validVLAN.

Switch RB3 460 performs the following processing:

1. Trap to CP (because it is a FIP packet)

2. CP generates FIP VLAN Notification (because it is acting as the FCFentity)

The FIP Notification flows from switch RB3 460 to FCoE host H1 474 asshown by arrow B 804. The FIP Notification packet has the followingvalues: DA: H1 ENode MAC, SA: Ext FCF MAC and VID: Same as Request.

For FIP Discovery, the FIP Solicitation packet flows from FCoE host H1474 to switch RB3 460 as shown by arrow A 802. The FIP Discovery packethas the following values: DA: ALL_FCFs_MAC, SA: H1 ENode MAC and VID:VCS FCoE VLAN.

Switch RB3 460 performs the following processing:

1. Trap to CP (because it is a FIP packet)

2. CP generates FIP Solicited Advertisement (because it is acting as theFCF entity)

The FIP Solicited Advertisement packet flows from switch RB3 460 to FCoEhost H1 474 as shown by arrow B 804. The FIP Solicited Advertisementpacket has the following values: DA: H1 ENode MAC, SA: Ext FCF MAC andVID: FCoE VLAN.

FIG. 9 illustrates the FIP FLOGI or FDISC operations. The FIP VirtualLink Instantiation (VLI) Login (FLOGI or FDISC) operations are asfollows. Assume that FCOE host H1 474 is logging into FC Domain 11, Port2 on switch 468. The FIP VLI Request packet flows from FCoE host H1 toswitch RB3 460 as shown by arrow A 902. The FIP VLI Request packet hasthe following values: DA: Ext FCF MAC, SA: H1 ENode MAC, VID: FCoE VLAN,DID: FFFFFE and SID: 000000.

The switch RB3 460 performs the following processing:

1. Trap to CP (because it is a FIP packet as determined by an FIPEthernet type trap)

2. Use Login Management as described below to determine egress RB forthe packet

3. TRILL forward to egress RB (normal TRILL forwarding)

The TRILL packet flows from switch RB3 460 to switch G-RB2 458 as shownby arrow B 904. The TRILL packet has the following values: TRILLDA=G-RB2, TRILL SA=RB3, Inner DA=G-RB2 int FCF MAC, Inner SA=H1 ENodeMAC, VID=FCoE VLAN, DID: FFFFFE and SID: 000000.

As switch G-RB2 458 is acting as the FFG functionality, it performs thefollowing login processing:

1. Normal FCoE-FC gateway FLOGI/FDISC processing (with the switch G-RB2458 presenting as an FC N_Port, most preferably a port that has NPIV orVirtual N_Port capabilities). In this case the switch G-RB2 458 convertsthe FIP FLOGI to an FC FDISC.

Thus the FDISC flows from the switch G-RB2 458 to switch 468 (FC Domain11) as shown by arrow C 906.

FC switch 468 responds to the FC FDISC from switch G-RB2 458 byreturning either an FC LS-ACC (successful login) or FC LS-RJT as shownby arrow D 908.

Switch G-RB2 458 performs the following response processing:

1. Normal FCoE-FC AG FLOGI/FDISC processing.

In this case the switch G-RB2 458 creates the FCoE FIP Response packetand provides it to switch G-RB3 460 as shown by arrow E 910. The TRILLpacket has the following values: TRILL DA=RB3, TRILL SA=RB2, Inner DA=H1Enode MAC, Inner SA=RB2 Int FCF MAC, VID=FCoE VLAN, DID=H1 FCID,SID=FFFFFE

Switch RB3 460 performs the following response processing:

1. Trap to CP (because it is a FIP packet as determined by an FIPEthernet type trap)

2. Decode response

3a: If Link Service Accept (LS ACC), send eNS update to all RBs

If the response is an LS_ACC, then switch RB3 460 provides an eNSnotification to all RBs as shown by arrows F 912.

All RBs perform the following response to the eNS notification:

3a.1. Normal MAC address update processing

3a.2. New FCoE VN_Port processing. Each RB adds an FCoE EXM entry usinglow order 3 bytes of VN_Port MAC address as FCID to indicate presence ofENode on the Ethernet Fabric 452.

4. Forward response to H1

The FIP VLI Response flows from switch RB3 460 to ENode host H1 474 asshown by arrow G 914. This FCoE FIP packet has the following values: DA:H1 ENode MAC, SA: Ext FCF MAC, VID: FCoE VLAN, DID=H1 FCID, SID=FFFFFE.

In a deployment where the G-RB is not at the first Ethernet Fabric hop,the first hop RB must decide to which G-RB it should forward FCoE Enodedevice FIP FLOGIs. In one embodiment an automatic method is used. In asecond embodiment, a configurable FCoE login management model is used.

The automatic embodiment utilizes the eNS distribution services toprovide updates from G-RBs to all other RBs with current EthernetFabric-FC link usage information. If the G-RB is the first hop, then ifa single logical FC N_Port to F_Port link (Ethernet Fabric-FC link) isestablished on the G-RB, forward the FC FLOGI on it. If multiple logicalEthernet Fabric-FC links are established on this G-RB, forward anEthernet Fabric FLOGI on the Ethernet Fabric-FC link with the leastnumber of FCoE device logins per link bandwidth. If a G-RB is not thefirst hop, then send the FLOGI to the G-RB with the least number ofEthernet Fabric-FC logins per total uplink bandwidth. Note that FDISCbased login from the same ENode must be sent to the same G-RB andforwarded on the same Ethernet Fabric-FC link, so the ENode to G-RBassociation must be retained in the first hop non-G-RB.

The second embodiment provides the SAN administrator with a facility tomap ENode devices to a specific Ethernet Fabric-FC link, and thereforeto a specific FC SAN Domain, Area. The resulting login managementdatabase is distributed to all RBs in the Ethernet Fabric.

By embedding FCoE to FC gateway functionality into the switches orRBridges that form an Ethernet Fabric the entire Ethernet Fabric can actas a virtual gateway, providing VF_Ports to the ENodes and N_Ports tothe FC fabric, and can route FCoE packets directly between connectedENode devices without a trip to the FC fabric.

If operation of an FCoE-only network without connection to an FC fabricis desired, in a first embodiment shown in FIG. 10, the FIP conversionand forwarding functions of the G-RB can be provided by an FC servicesdaemon 1014 provided in an F-RB1 1004 or FC services appliance 1034, theFC services daemon 1014 or FC services appliance 1034 providing allnecessary FC operations, including the Name Server function, to allowENodes 1006 to receive FCIDs. FIP VLI Login operations received at theRBs 1004, such as RB2 and RB3, are directed to the F-RB3 1004 containingthe FC services daemon 1014 or the FC services appliance 1034 instead ofan N_Port of a G-RB. This allows the RBs to develop the eNS entries forthe ENodes to allow direct routing of the FCoE packets such as path 1012in this FCoE-only, no FC fabric embodiment.

In a second alternate embodiment illustrated in FIG. 11, each RB 1104 inthe Ethernet Fabric 1102 includes an FCF daemon 1114 (FCFD) and so isreferred to as an F-RB. The FCF daemon 1114 provides all FC servicesnormally provided by an FCF, which is sufficient to operate an FCfabric. In this embodiment, rather than each RB 1104 forwarding FIP VLIoperations to another RB 1104 or an FC services appliance, each RB 1104handles the login operations of its connected ENodes 1106 as it containsfull FC services. eNS can be used as above to maintain the tables usedfor direct routing such as path 1112 or similar FC inter-switchcommunications can be used, as those will be provided to allow the FCFdaemons 1114 to operate as a normal FC fabric.

While certain specific embodiments of particular functions have beenmentioned, in most cases other similar embodiments that perform the samefunction can be used. For example, eNS is described as providing messagedistribution services between the Ethernet Fabric switches but othermessage distribution methods and protocols could be used. As anotherexample, the GD is described as a user space module, language usuallyused with Linux/Unix variants, but it could be a kernel module inLinux/Unix or could be the equivalent in Windows, VxWorks and the like.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described embodiments may be used incombination with each other. Many other embodiments will be apparent tothose of skill in the art upon reviewing the above description. Thescope of the invention should, therefore, be determined with referenceto the appended claims, along with the full scope of equivalents towhich such claims are entitled. In the appended claims, the terms“including” and “in which” are used as the plain-English equivalents ofthe respective terms “comprising” and “wherein.”

The invention claimed is:
 1. An Ethernet Fabric comprising: a pluralityof interconnected switches, at least one of said plurality ofinterconnected switches for connection to a Fibre Channel over Ethernet(FCoE) ENode, at least one of said plurality of interconnected switchesfor connection to a Fibre Channel (FC) fabric, said at least one FCfabric-connected switch for connection to the FC fabric with an N_Portand including FCoE to FC gateway functionality, each of said pluralityof interconnected switches including a table for storage of all ENodesconnected to the Ethernet Fabric and all N_Ports connected to the FCfabric, wherein all switches for connection to ENodes trap any FCoEInitialization Protocol (FIP) F Port Login (FLOGI) or (Discover F PortParameters (FDISC) Link Service Accept (LS_ACC) packet directed to aconnected ENode, add a table entry of the connected ENode and inform allother switches in the Ethernet Fabric of the connected ENode address,wherein all switches for connection to the FC fabric log into the FCfabric, add a table entry of the connected N_Port and inform all otherswitches of the FC domain and area of the connected N_Port, wherein anyswitch connected to an ENode routes FCoE packets to the proper N_Portbased on domain and area values in said table, wherein any switchconnected to the FC fabric routes packets received from the FC fabric tothe proper ENode based on addresses of ENodes in said table, and whereinany switch connected to an ENode routes FCoE packets to the proper ENodebased on addresses of ENodes in said table.
 2. The Ethernet Fabric ofclaim 1, wherein any switch connected to an ENode generates responses toFIP Discovery requests and forwards FIP Virtual Link Instantiation (VLI)requests to an entity which provides FC addresses.
 3. The EthernetFabric of claim 2, wherein the entity which provides FC addresses is anFC switch in the FC fabric so that the FIP VLI packets are routed to aswitch connected to the FC fabric.
 4. An Ethernet Fabric switchcomprising: a control processor; an Ethernet port configured for FibreChannel over Ethernet (FCoE) operation for connection to an ENode; anFibre Channel (FC) port configured for connection to an FC fabric as anN_Port; an Ethernet Fabric port for connection to another EthernetFabric switch to allow development of an Ethernet Fabric; aninterconnect to connect said control processor, said Ethernet port, saidEthernet Fabric port and said FC port; and routing table storage coupledto said control processor, said Ethernet port, said Ethernet Fabric portand said FC port, wherein said control processor, said Ethernet port,said Ethernet Fabric port and said FC port cooperate to act as an FCoEto FC gateway, wherein said control processor, said Ethernet port, saidEthernet Fabric port and said FC port cooperate as needed to trap anyFCoE Initialization Protocol (FIP) F Port Login (FLOGI) or (Discover FPort Parameters (FDISC) Link Service Accept (LS_ACC) packet directed toa connected ENode, add a table entry of the connected ENode and informany other switches in the Ethernet Fabric of the connected ENodeaddress, wherein said control processor, said Ethernet Fabric port andsaid FC port cooperate as needed to log into the FC fabric, add a tableentry of the connected N_Port and inform all other switches in theEthernet Fabric of the FC domain and area of the connected N_Port,wherein said control processor, said Ethernet port, said Ethernet Fabricport and said FC port cooperate as needed to route FCoE packets to theproper N_Port based on domain and area values in said table, whereinsaid control processor, said Ethernet port, said Ethernet Fabric portand said FC port cooperate as needed to route packets received from theFC fabric to the proper ENode based on addresses of ENodes in saidtable, and wherein said control processor, said Ethernet port and saidEthernet Fabric port cooperate as needed to route FCoE packets to theproper ENode based on addresses of ENodes in said table.
 5. The EthernetFabric switch of claim 4, said control processor, said Ethernet port,said Ethernet Fabric port and said FC port cooperate as needed togenerate responses to FIP Discovery requests and forward FIP VirtualLink Instantiation (VLI) requests to the entity which provides FCaddresses.
 6. The Ethernet Fabric switch of claim 5, wherein the entitywhich provides FC addresses is an FC switch in the FC fabric so that theFIP VLI packets are routed to an Ethernet Fabric switch connected to theFC fabric.
 7. An Ethernet Fabric comprising: a plurality ofinterconnected switches, at least one of said plurality ofinterconnected switches for connection to a Fibre Channel over Ethernet(FCoE) ENode, each of said plurality of interconnected switchesincluding a table for storage of all ENodes connected to the EthernetFabric, wherein all switches for connection to ENodes trap any FCoEInitialization Protocol (FIP) F Port Login (FLOGI) or (Discover F PortParameters (FDISC) Link Service Accept (LS_ACC) packet directed to aconnected ENode, add a table entry of the connected ENode and inform allother switches in the Ethernet Fabric of the connected ENode address,and wherein any switch connected to an ENode routes FCoE packets to theproper ENode based on addresses of ENodes in said table.
 8. The EthernetFabric of claim 7, wherein any switch connected to an ENode generatesresponses to FIP Discovery requests and forwards FIP Virtual LinkInstantiation (VLI) requests to an entity which provides FC addresses.9. An Ethernet Fabric switch comprising: a control processor; anEthernet port configured for Fibre Channel over Ethernet (FCoE)operation for connection to an ENode; an Ethernet Fabric port forconnection to another Ethernet Fabric switch to allow development of anEthernet Fabric; an interconnect to connect said control processor, saidEthernet port and said Ethernet Fabric port; and routing table storagecoupled to said control processor, said Ethernet port and said EthernetFabric port, wherein said control processor, said Ethernet port and saidEthernet Fabric port cooperate as needed to trap any FCoE InitializationProtocol (FIP) F Port Login (FLOGI) or (Discover F Port Parameters(FDISC) Link Service Accept (LS_ACC) packet directed to a connectedENode, add a table entry of the connected ENode and inform any otherswitches in the Ethernet Fabric of the connected ENode address, andwherein said control processor, said Ethernet port and said EthernetFabric port cooperate as needed to route FCoE packets to the properENode based on addresses of ENodes in said table.
 10. The EthernetFabric switch of claim 9, said control processor, said Ethernet port andsaid Ethernet Fabric port cooperate as needed to generate responses toFIP Discovery requests and forward FIP Virtual Link Instantiation (VLI)requests to an entity which provides FC addresses.
 11. A method ofoperating an Ethernet Fabric comprising: interconnecting a plurality ofswitches, at least one of said plurality of interconnected switches forconnection to a Fibre Channel over Ethernet (FCoE) ENode, at least oneof said plurality of interconnected switches for connection to a FibreChannel (FC) fabric, said at least one FC fabric-connected switch forconnection to the FC fabric with an N_Port and including FCoE to FCgateway functionality, each of said plurality of interconnected switchesincluding a table for storage of all ENodes connected to the EthernetFabric and all N_Ports connected to the FC fabric, trapping any FCoEInitialization Protocol (FIP) F Port Login (FLOGI) or (Discover F PortParameters (FDISC) Link Service Accept (LS_ACC) packet directed to aconnected ENode by all switches for connection to ENodes, adding a tableentry of the connected ENode and informing all other switches in theEthernet Fabric of the connected ENode address, logging into the FCfabric by all switches for connection to the FC fabric, adding a tableentry of the connected N_Port and informing all other switches of the FCdomain and area of the connected N_Port, routing FCoE packets to theproper N_Port based on domain and area values in said table by anyswitch connected to an ENode, routing packets received from the FCfabric to the proper ENode based on addresses of ENodes in said table byany switch connected to the FC fabric, and routing FCoE packets to theproper ENode based on addresses of ENodes in said table by any switchconnected to an ENode.
 12. The method of claim 11, further comprising:generating responses to FIP Discovery requests by any switch connectedto an ENode; and forwarding FIP Virtual Link Instantiation (VLI)requests to an entity which provides FC addresses by any switchconnected to an ENode.
 13. The method of claim 12, wherein the entitywhich provides FC addresses is an FC switch in the FC fabric so that theFIP VLI packets are routed to a switch connected to the FC fabric.
 14. Amethod of operating an Ethernet Fabric switch comprising: acting as aFibre Channel over Ethernet (FCoE) to Fibre Channel (FC) gateway;trapping any FCoE Initialization Protocol (FIP) F Port Login (FLOGI) or(Discover F Port Parameters (FDISC) Link Service Accept (LS_ACC) packetdirected to a connected ENode; adding a routing table entry of theconnected ENode; informing any other switches in the Ethernet Fabric ofthe connected ENode address; logging into the FC fabric as an N_Port;adding a routing table entry of the connected N_Port; informing allother switches in the Ethernet Fabric of the FC domain and area of theconnected N_Port; routing FCoE packets to the proper N_Port based ondomain and area values in said routing table; routing packets receivedfrom the FC fabric to the proper ENode based on addresses of ENodes insaid routing table; and routing FCoE packets to the proper ENode basedon addresses of ENodes in said table.
 15. The method of claim 14,further comprising: generating responses to FIP Discovery requests; andforwarding FIP Virtual Link Instantiation (VLI) requests to an entitywhich provides FC addresses.
 16. The method of claim 15, wherein theentity which provides FC addresses is an FC switch in the FC fabric sothat the FIP VLI packets are routed to an Ethernet Fabric switchconnected to the FC fabric.
 17. A method of operating an Ethernet Fabriccomprising: interconnecting a plurality of interconnected switches, atleast one of said plurality of interconnected switches for connection toa Fibre Channel over Ethernet (FCoE) ENode, each of said plurality ofinterconnected switches including a table for storage of all ENodesconnected to the Ethernet Fabric; trap any FCoE Initialization Protocol(FIP) F Port Login (FLOGI) or (Discover F Port Parameters (FDISC) LinkService Accept (LS_ACC) packet directed to a connected ENode by allswitches for connection to ENodes, adding a table entry of the connectedENode and informing all other switches in the Ethernet Fabric of theconnected ENode address, and routing FCoE packets to the proper ENodebased on addresses of ENodes in said table by any switch connected to anENode.
 18. The method of claim 17, further comprising: generatingresponses to FIP Discovery requests by any switch connected to an ENode;and forwarding FIP Virtual Link Instantiation (VLI) requests to anentity which provides FC addresses.
 19. A method of operating anEthernet Fabric switch comprising: a control processor; an Ethernet portconfigured for Fibre Channel over Ethernet (FCoE) operation forconnection to an ENode; an Ethernet Fabric port for connection toanother Ethernet Fabric switch to allow development of an EthernetFabric; an interconnect to connect said control processor, said Ethernetport and said Ethernet Fabric port; and routing table storage coupled tosaid control processor, said Ethernet port and said Ethernet Fabricport, wherein said control processor, said Ethernet port and saidEthernet Fabric port cooperate as needed to trapping any FCoEInitialization Protocol (FIP) F Port Login (FLOGI) or (Discover F PortParameters (FDISC) Link Service Accept (LS_ACC) packet directed to aconnected ENode; adding a routing table entry of the connected ENode;informing any other switches in the Ethernet Fabric of the connectedENode address; and routing FCoE packets to the proper ENode based onaddresses of ENodes in said routing table.
 20. The method of claim 19,further comprising: generating responses to FIP Discovery requests; andforwarding FIP Virtual Link Instantiation (VLI) requests to an entitywhich provides FC addresses.